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What is GUR? 

• Combination of (evolving) methods to 
explore users' experiences of games 

• Much more than usability 

• Goal = collect unbiased data to make 
games better 
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Why is this important? 

• Games have to provide a good 
experience 

• Productivity tools = outcome is tangible 

• Games = outcome IS the experience 
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GUR Helps Games Succeed 



• " Halo 3: How Microsoft Labs 
Invented a New Science of Play" - 
Thompson (2007), Wired 


"Bungie fixed the terrain 
to keep players from 
backtracking" 


http ://www. wired. com/gaming/virtualworlds/magazine/15-09/ff_halo?currentPage=all 
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GUR is Growing 

• Big studios do it 

• Top games have no 
reported major usability 
issues 

• http://www.stevebromley.com/bloQ/ 
2014/02/24/how-important-is-aames- 
usability-to-reviewers/ 
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GUR Course 

• Aimed at Juniors (undergrads) 

• Co-designed with Jose Zagal 

• REQUIRED in the design track as of 2012 

• Pre-req: Basic stats + CITI cert by week 4 

• lx a week for 3-hours + 15 minute break 
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DePaul's Game Program 

• 315 - Undergraduate 

• 179 - Design 

• 126 - Programming 

• 10 - Non-declared 

• 53 MS Game Development 
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Takeaways 

• What you need to set up a course like 
this at your institution 

• Lab/Equipment 

• Topics & methods we cover 

• Highlight how to scaffold the students so they are 
conducting studies on their own 
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(actual students) 


(not actual student) 
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Overview 

• The basics 

• Equipment & labs 

• Game selection 

• Topics & methods we cover 

• Post mortem 
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Course objectives 

• Understand several methods used to 
evaluate games 

• Strengths/weaknesses of each 

• When to use them in the dev. process 

• Hands-on experience implementing 
methods 
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• Perform common GUR methods: 

• Competitive Review 

• Heuristic Review 

• Usability 

• GUR Playtesting 

• Present findings effectively 
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Overview 

• 'he basics 

• Equipment & Labs 

• Game selection 

• Topics & methods we cover 

• Post mortem 
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Our labs - 
downtown Chicago 
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Playtesting - 7 th floor 
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http://download.techsmith.com/morae/docs/onlinehelp/3.1/Morae-ManagerHelp.pdf 
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Playtesting station (Multiple needed) 

Approx, cost 

Monitor (24") 

$175.00 

Mac Mini w/ Morae Recorder (boots as a PC) 

$800.00 

Gaming system (Xbox) 

$500.00 

Keyboard, Mouse, HDMI switch 

$150.00 

Cables, power strip, misc (extra batteries) 

$30.00 

Headphones (not shown) 

$30.00 

Total $1685.00 
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Usability 


Usability station with Morae Recorder 

IP Address — - 
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Usability Lab - 8 th floor 
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Usability recording station (One needed) 

Approx, cost 

Monitor (Apple Thunderbolt) 

$1000.00 

Mac tower with Morae manager and observer 
(boots as a PC) 

$4500.00 

Keyboard, Mouse 

$100.00 

For usability station add game capture +cable 

$200.00 

Total 

$5800.00 


Total = $22,650 with 10 playtest stations 
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Overview 

• 'he basics 

• Game selection 

• Week by week 

• Post mortem 
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Games? 

• Commercial? 

• Indie? 

• Student projects? 
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Tips for Selecting Games 

• Availability: Many copies (10 in our case) 

• Price: Inexpensive (Under $10 each, used) 

• Stable build 

• Low ratings (lots of problems to find) 

• Short tutorial/intro cut-scenes 

• Does not require specialized knowledge 
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Worked really well 



® XBOX 360 



XBOX 360 



© 


XBOX 360 HU lIve 


WI «1 

EXTREME CONDITION 


0 XBOX 360 uve 
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Okay ...but too good of game 




MEDALtrt^ 

HONOR 

WARFIGHTER 



©XBOX 360 
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Too much special knowledge 
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Overview 


• Topics & methods we cover 

• Post mortem 
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Topics & methods we cover 

• Course has three parts 

• Part 1: Introduction, early methods 

• Part 2: Usability 

• Part 3: GUR Playtesting 
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Part One (three weeks) 

• You are not your user 

• Competitive Review 

• Heuristic evaluation 
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You Are Not Your User 

• Keirsey Temperament test 

• Bateman's DGD1 

. Conqueror (TJ) - all ISTJ, INTJ, ESTJ & ENTJ 
. Manager (TP) - ISTP, INTP, ENTP & ESTP 
. Wanderer (FP) - INFP, ENFP, ISFP & ESFP 
. Participant (FJ) - ESFJ, ISFJ, ENFJ & INFJ 
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Competitive Review 

1. Identify and profile the competition 

2 . Define a set of key dimensions for 
comparison 

3. Compare competitors to each other 

4. Use the comparisons to create 
recommendations for action 
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• Student example (summary table) 

Rating*: 1 (very poor) to 5 (excellent) 


Game 

Racing 

Game Modes 

Graphics 

Physics 

Environment 

Multiplayer 

Baja: Edge 
of Control 

3 

3 

4 

4 

4 

3 

DiRT 

3 

3 

5 

5 

4 

3 

Burnout: 

Paradise 

4 

4 

5 

5 

4 

4 

Hot Wheels: 
Beat That! 

4 

3 

2 

3 

3 

3 
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Heuristic Evaluation 

• Expert reviews (assess problems based on 
known lists) 

• Ideal = five evaluators and then compare lists 

• Heuristics for games need to focus on 
different attribute than those for 
productivity tools... 
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Heuristics for games 

• For example: pace, game mechanics, 
heads up display, memory load for users, 
and gameplay, etc. 

• We discuss several: 

• Federoff: Interface, Mechanics and Play 

• HEP (Heuristic Evaluation for Playability): Play, Mobility 
and Usability 

• Schaffer: General, GUI, Play 
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• Student example 

(presentation format from Isbister and Schafer) 


Heuristic 

Provide clear goals, present overriding goal early as well as short-term goals 
throughout play. 

Problem 

The game does not provide clear goals. 

Rating 

Important 

Description 

The game describes what are the possible goals. However, to find what the current goal 
players must search for it on a cluttered screen. 

Solution 

Clearly show what goal is before a race begins. 
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Topics & methods we cover 

• Three parts 

• Part 1: Introduction, early methods 

• Part 2: Usability 

• Part 3: GUR Playtesting 
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Part Two: Usability (three weeks) 

• In the context of games = about player 
behavior and understanding 

• Can the player equip their weapon 

• Does the player understand how to complete 
the level 

• Assessment (summative) - (task driven) 

• Think aloud protocol 



GDC EDUCATION SUMMIT 


GAME DEVELOPERS CONFERENCE® 2014 MARCH 17-21, 2014 GDCONF.COM 


Usability Plan & Moderator Guide 


• Scaffolding the 
writing & planning 

• Adapted from Rubin & 
Chisnell 


Jeffrey Rubin and Dana Chisnell 


Foreword by Jared Spool 


Handbook of 

Usability 

Testing 



How to 
Plan, Design, 
and Conduct 
Effective Tests 


S E C 0 N Qu«ed E™D I T I 0 N 
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You need to delete or edit anything that is in red text 
[Game Title] 

Appendix A: Usability test plan and Moderator guide 


Authors: 
Published: 
Study type: 

List your names 

Date of the report 

Genre: 

Game genre 

Usability 

Platform: 

X-Box 360 


1.0 Usability test plan 

In the following sections, we describe the purpose of our usability test (section 1.1), our research 
questions (section 1.2), the measures we will collect (section 1,3), recruitment parameters (section 
1.4) and outline our procedure (section 1.4). 

1.1: Purpose for the test 

- summarize in a paragraph ortwo what is the purpose of yourtest. 

For example from the Poker team: 

In this usability study, we will assess the usability of the interface for users. Specifically, we will 
explore if users can (1) create a new career, (2) complete the introductory level, (3) select a 
tournament, (4) and complete the selected tournament. 

1.2: Research questions 

-list your specific research questions. 

Example from Poker team: 

(1) Can users create a new career? 

(2) Can users raise, call, fold, and go all in? 

(3) Can users check their cards? 

1.3 Measures 

These are what you are going to pay attention to - summarize briefly here. 
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Usability Report 

• Adapted from GUR at Microsoft 

• Again. ..a template 

• Scaffold the writing so they can focus on 
conducting the studies 
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Usablity report 


Authors: 

Your names 

Published: 

Date 

Genre: 

Do this 

Study type: 

Usability 

Platform: 

X-Box 360 


Report Summary 

Summarize what part of the game you were looking at. Include your research questions here. 

List how many participants you had, their ages and gender - and briefly describe them (their 
experience level ...etc.) 

Highlights 

• Finding 1: 

Summarize in one line: These are things that went well - e.g. all participants were able to find 
and equip weapons 

• Finding 2: 

Summarize in one line: These are things that went well - e.g. all participants were able to find 
and equip weapons 

Lowlights 

• Issue 1: 

Summarize in one line: These are things that did not go well - e.g. two participants did not 
understand how to perform a jump in Skate 

• Issue 2: 

Summarize in one line: These are things that did not go well - e.g. two participants did not 
understand how to perform a jump in Skate 

Top Action Items 

• Recommendation 1: 

These are top things you recommend that the game makers fix- these come directly from the 
action items below. 

• Recommendation 2: 

These are top things you recommend that the game makers fix- these come directly from the 
action items below. 


Highlights 

Finding 1: Title 

Provide more detail, e.g. After practice with the prototypes, all 5 participants were able to scroll from 
one end of the car list to the other. Moreover, all participants were able to move in between a few cars 
at a time in the list when asked to do so. 

Include screen caps if needed - include graphs as needed 

"Indue a relevant quotes from a participants" - Participant 4 

Action Items: 

None, this is good. (Sometimes, even a highlight will need on action) 


Finding 2: Title 

Here is an example from the Poker team: 

After loading up the game for the first time, participants were prompted to start the career mode. All of 
them were able to navigate the first menu and begin the career without any problems 

Include screen caps if needed - include graphs as needed 

"Indue a relevant quotes from a participants" - Participant 4 


Action Items: 

None, this is good. (Sometimes, even a highlight will need an action) 


Add more findings as needed 
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Topics & methods we cover 

• Three parts 

• Part 1: Introduction, early methods 

• Part 2: Usability 

• Part 3: GUR Playtesting 
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Part 3: GUR Playtests 

• Participants' attitudes and opinions 

• Our method 

• Players play for a period of time or to a 
specific goal (about 20 minutes) 

• Stop - take a questionnaire (flip HDMI switch) 

• Repeat 3x for an hour of playtime 
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Research Q v. Playtest Q 

• RQ - What the researcher wants to know 

• What did users think of the graphics? 
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What the Participant Sees 

• Playtest Q closed 

• How satisfied were you with the artistic style 
of the graphics? 

.(1) Very dissatisfied- (3)Neither satisfied or 
dissatisfied - (5) Very satisfied 

• Playtest Q open 

• What did you LIKE MOST about the graphics? 

• What did you LIKE LEAST about the graphics? 
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Participant Samples 

• Usability = 5-7 v. GUR Playtest = 30 
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Topics & methods we cover 

• Three parts 

• Part 1: Introduction, early methods 

• Part 2: Usability 

• Part 3: GUR Playtesting 
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Overview 

• 'he basics 

• 'opics & methods we cover 

• Post mortem 
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If only... 

• Ideal = 15 weeks (semester) 

• Have them evaluate a peer's capstone project 

• Recruitment is difficult 

• Set-up with instructors before term 

• We will be setting up a participant pool for 
2014-2015 for all classes that involve 
participants 
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...more 

• Class cap (20) ...ideal around 15 (group 
size = 3) 

• First time (winter 2013) = 13 

• This time (winter 2014) = 32 

• We are now running two sections a year 

• Students really need a lot of scaffolding in 
the writing 

• Added the templates this year 
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Takeaways 

• What you need to set up a course like 
this at your institution 

• Lab/Equipment 

• Topics & methods we cover 

• Highlight how to scaffold the students so they are 
conducting studies on their own 
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Questions? 


• http://facsrv.cs.depaul.edu/~cputnam/protected/classes/ 
GAM312 winter!4/schedule.p h p 


• user name = student 

• password = lh4veAccess 

• cputnam@cdm.depaul.edu 

• http://www.cyputnam.com/ 




